{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Kubeflow pipelines\n",
    "\n",
    "This notebook goes through the steps of using Kubeflow pipelines using the Python3 interpreter (command-line) to preprocess, train, tune and deploy the babyweight model.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Start Hosted Pipelines and Notebook\n",
    "\n",
    "To try out this notebook, first launch Kubeflow Hosted Pipelines and an AI Platform Notebooks instance.\n",
    "Follow the instructions in this [README.md](pipelines/README.md) file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Install necessary packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --quiet kfp python-dateutil --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure to *restart the kernel* to pick up new packages (look for button in the ribbon of icons above this notebook)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Connect to the Hosted Pipelines\n",
    "\n",
    "Visit https://console.cloud.google.com/ai-platform/pipelines/clusters\n",
    "and get the hostname for your cluster.  You can get it by clicking on the Settings icon.\n",
    "Alternately, click on the Open Pipelines Dashboard link and look at the URL.\n",
    "Change the settings in the following cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CHANGE THESE\n",
    "PIPELINES_HOST='447cdd24f70c9541-dot-us-central1.notebooks.googleusercontent.com'\n",
    "PROJECT='ai-analytics-solutions'\n",
    "BUCKET='ai-analytics-solutions-kfpdemo'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import kfp\n",
    "import os\n",
    "client = kfp.Client(host=PIPELINES_HOST)\n",
    "#client.list_pipelines()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. [Optional] Build Docker containers\n",
    "\n",
    "I have made my containers public (See https://cloud.google.com/container-registry/docs/access-control on how to do this), so you can simply use my images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "cd pipelines/containers\n",
    "#bash build_all.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check that the Docker images work properly ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!docker run -t gcr.io/ai-analytics-solutions/babyweight-pipeline-bqtocsv:latest --project $PROJECT  --bucket $BUCKET --local"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Upload and execute pipeline\n",
    "\n",
    "Upload to the Kubeflow pipeline cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Experiment link <a href=\"http://447cdd24f70c9541-dot-us-central1.notebooks.googleusercontent.com/#/experiments/details/305492c2-1ff3-46e4-b9bf-3c63790eb46a\" target=\"_blank\" >here</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run link <a href=\"http://447cdd24f70c9541-dot-us-central1.notebooks.googleusercontent.com/#/runs/details/5c026be2-a2b4-4135-a604-17fe4f09a107\" target=\"_blank\" >here</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pipelines.containers.pipeline import mlp_babyweight\n",
    "\n",
    "args = {\n",
    "    'project' : PROJECT, \n",
    "    'bucket' : BUCKET\n",
    "}\n",
    "\n",
    "#pipeline = client.create_run_from_pipeline_func(mlp_babyweight.preprocess_train_and_deploy, args)\n",
    "\n",
    "os.environ['HPARAM_JOB'] = 'babyweight_200207_231639' # change to job from complete step\n",
    "pipeline = client.create_run_from_pipeline_func(mlp_babyweight.train_and_deploy, args)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2020 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#      http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
